Skip to content

[UX] Show No quota and No balance when provisioning fails - #4372

Open
peterschmidt85 wants to merge 1 commit into
masterfrom
issue_496_quota_balance_provisioning
Open

peterschmidt85 wants to merge 1 commit into
masterfrom
issue_496_quota_balance_provisioning

Conversation

@peterschmidt85

@peterschmidt85 peterschmidt85 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Offers can show No quota / No balance while provisioning reports No offers / No capacity. In dstack Factory (including Sky), this makes quota and billing restrictions look like provider capacity problems.

Steps to reproduce

  1. Use a Factory/Sky project with a dstack backend, positive owner balance, no reusable instances, and available Nebius H100 offers matching cpu: 8... Ensure no other matching offers are usable.

  2. Set the project's CPU quota to 4; matching offers should show no_quota.

  3. Apply this fleet.yml:

    type: fleet
    name: quota-repro
    nodes: 0..1
    backends: [nebius]
    resources:
      cpu: 8..
      gpu: H100
  4. Apply this task.yml, confirm submission, and inspect the failed run/job:

    type: task
    name: quota-task
    fleets: [quota-repro]
    retry: false
    resources:
      cpu: 8..
      gpu: H100
    commands:
      - echo hello
  5. To exercise fleet-instance provisioning, apply another fleet with the same resources, a different name, and nodes: 1; inspect its terminated instance.

  6. Repeat with sufficient quota and zero owner balance for no_balance. Use another available provider/GPU if needed.

Actual behaviour

Object Reported failure
Offer No quota / No balance
Fleet instance No offers
Job/run No capacity

Expected behaviour

Show No quota / No balance when no usable offers remain and all matching rejected offers share that reason. Keep the generic failure for mixed blockers or failed provisioning attempts.

CLI status text:

no quota
no balance

Notes

Availability filtering currently discards rejected offers before provisioning can inspect their reason. get_offers_by_requirements() now returns an Offers result containing selected offers and an optional common rejection reason, using the same matching rules and one lookup.

Backward compatibility

For rolling deployments, keep storing the existing job reason FAILED_TO_START_DUE_TO_NO_CAPACITY and instance reason NO_OFFERS. Put No quota / No balance in the existing termination_reason_message. Older replicas can read these rows; their status text remains generic.

New enums and their status/retry/cleanup handling are included, but the new values are not emitted. TODOs defer emission until a future release establishes a minimum supported upgrade version that handles them on every replica.

@peterschmidt85 peterschmidt85 changed the title Preserve quota and balance provisioning failures [UX] Show No quota and No balance when provisioning fails Oct 14, 2026
@peterschmidt85
peterschmidt85 marked this pull request as ready for review October 16, 2026 08:09
@r4victor

Copy link
Copy Markdown
Collaborator

Deployment: during a rolling update, old and new servers share the DB. If a new server saves one of the new job failure reasons, an old server fails when reading that job because it does not recognize the value. Safe rolling updates need two releases: first teach every server to read and handle these reasons without saving them; then start saving them. This draft must be split into those two releases before deployment.

Is this addressed?

@peterschmidt85
peterschmidt85 marked this pull request as draft October 16, 2026 10:36
@peterschmidt85
peterschmidt85 force-pushed the issue_496_quota_balance_provisioning branch from a79f3ec to ce3a87b Compare October 17, 2026 11:48
@peterschmidt85
peterschmidt85 marked this pull request as ready for review October 17, 2026 11:48
@peterschmidt85

Copy link
Copy Markdown
Contributor Author

@r4victor yes, addressed in ce3a87b:

  • Rolling deployment: retain the existing stored reasons and put No quota / No balance in termination_reason_message, so old replicas can read the rows. New enums have read/retry/cleanup support but are not emitted; TODOs defer emission until the minimum supported upgrade version handles them on every replica.
  • Callback: removed. Offer selection returns selected offers and an optional common rejection reason in an Offers result from the same lookup. Provisioning consumes that result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants